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Timeline of recent CPU flaws, 2018 (a) 


Jan 03 
Jan 03 
Jan 03 


May 21 


Jun 21 


Spectre v1: Bounds Check Bypass 
Spectre v2: Branch Target Injection 
Meltdown: Rogue Data Cache Load 


Spectre-NG: Speculative Store 
Bypass 


TLBleed: Side-channel attack over 
shared TLBs 


(Ə) redhat 
Timeline of recent CPU flaws, 2018 (b) 
Jun 29 » NetSpectre: Side-channel attack 
over local network 


Jul 10 4 Spectre-NG: Bounds Check Bypass 
Store 


Aug 14 8 11ТЕ: "(1 Terminal Fault" 
? 
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(Ə) redhat 
What this talk is not about 


Out of scope: 
Internals of various side-channel attacks 
How to exploit Meltdown & Spectre variants 


Details of performance implications 
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Out of scope: 
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лә Related talks in the ‘References’ section 
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KVM-based virtualization components 


Linux with KVM 
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OpenStack, 
et al. 
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KVM-based virtualization components 


OpenStack, 
et al. 














libguestfs 
(guestfish) 


Virt Driver 


libvirtd 
2 Q 


A блә 


Custom 





Appliance 





Linux with KVM 


(Ə) redhat 
QEMU and KVM 


siza 


ioct10 — /dev/kvm 


[kvm.ko; kvm-intel.kol 
VMX modes: guest++host 
Emulation: CPUID, irqchip 


VMLAUNCH, ... 


Hardware: Intel VMX extensions 





(Ə) redhat 
QEMU and KVM 


oa 






ioct10 — /dev/kvm 


To inspect, use 


Linux tools: [kvm.ko; kvm-intel.kol 
ка VMX modes: guest++host 
Emulation: CPUID, irqchip 


VMLAUNCH, ... 


Hardware: Intel VMX extensions 


O redhat. 


Hardware-based virtualization with KVM 






KVM prepares 
to enter CPU 
Guest Mode 


Perform in-kernel 
emulation 


VMENTER 









QEMU issues 
ioct1(KVM RUN) 
QEMU emulates 

hardvvare 


Execute natively 
in Guest Mode. 
(CPU with VMX) 





VMEXIT 


Emulate 


in-kernel? 


No 
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Part | 
Interfaces to configure vCPUs 
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(Ə) redhat 
x86: QEMU”s default CPU models (a) 


The default models (qemu32, qemu64) work on any host CPU 
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(Ə) redhat 
x86: QEMU”s default CPU models (a) 


The default models (qemu32, qemu64) work on any host CPU 


But they are dreadful choices! 
No AES / AES-NI: critical for TLS performance 
No RDRAND: important for entropy 
No PCID: performance- & security-critical (thanks, Meltdown) 


(Ə) redhat 
x86: QEMU”s default CPU models (b) 


$ cd /sys/devices/system/cpu/vulnerabilities/ 

$ grep . * 

litf:Mitigation: PTE Inversion 
meltdown:Mitigation: PTI 
spec_store_bypass:Vulnerable 
spectre_vi:Mitigation: __user pointer sanitization 
spectre_v2:Mitigation: Full generic retpoldne 
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(Ə) redhat 
x86: QEMU”s default CPU models (b) 


$ cd /sys/devices/system/cpu/vulnerabilities/ 


$ GS | Ona guest running with qemu64 
litf:Mitiga 


meltdown:Mitigation: PTI 
spec_store_bypass:Vulnerable 
spectre_vi:Mitigation: __user pointer sanitization 
spectre_v2:Mitigation: Full generic retpoldne 
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(Ə) redhat 
x86: QEMU”s default CPU models (b) 


$ cd /sys/devices/system/cpu/vulnerabilities/ 
$ grep . * 

litf:Mitigation: PTE Inversion 
meltdown:Mitigation: PTI 

spec_store_bypass: Vulnerable 


spectre vl Spectre-NG user pointer Sanitization 
spectre_v2:; 5 11 generic retpoline 
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(Ə) redhat 
x86: QEMU”s default CPU models (b) 


$ cd /sys/devices/system/cpu/vulnerabilities/ 

$ grep . * 

litf:Mitigation: PTE Inversion 
meltdown:Mitigation: PTI 

spec store bypass:Vulnerable 
spectre_vi:Mitigation: user pointer sanitization 
spectre v2:Mitigation: Full generic retpoline 


~> Always specify an explicit CPU model; 
or use libvirt’s host-model 
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Defaults of other architectures 


AArch64: Doesn't provide a default guest CPU 


$ qemu-system-aarch64 -machine virt -cpu help 
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AArch64: Doesn't provide a default guest CPU 


$ qemu-system-aarch64 -machine virt -cpu help 


Default CPU depends on 
the machine type 
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Defaults of other architectures 


AArch64: Doesn't provide a default guest CPU 


$ qemu-system-aarch64 -machine virt -cpu help 


ppc64 — host for KVM; power8 for TCG (pure emulation) 
s390x — host for KVM; qemu for TCG 
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Configure CPU on the command-line 


On x86, by default, the qemu64 model is used: 


$ qemu-system-x86_64 [...] 
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Configure CPU on the command-line 


On x86, by default, the qemu64 model is used: 
$ qemu-system-x86_64 [...] 


Specify a particular CPU model: 
$ qemu-system-x86_64 -cpu IvyBridge-IBRS [...] 


Named CPU model 


12/38 


(Ə) redhat 
Control guest CPU features 


Enable or disable specific features for a VCPU model: 


$ qemu-system-x86 64 \ 
-cpu Skylake-Client-IBRS, vmx=off,pcid=on [...] 
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(Ə) redhat 
Control guest CPU features 


Enable or disable specific features for a VCPU model: 


$ qemu-system-x86 64 \ 
-cpu Skylake-Client-IBRS,vmxeoff,pcideon [...] 


Granular CPU flags 
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су redhat. 


Control guest CPU features 


Enable or disable specific features for a VCPU model: 


$ qemu-system-x86 64 \ 
-cpu Skylake-Client-IBRS, vmx=off,pcid=on [...] 


For a list of supported vCPU models, refer to: 
$ qemu-system-x86 64 -cpu help 


Or libvirt’s — ‘virsh cpu-models x86 64” 
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(Ə) redhat 
QEMU”s CPU-related run-time interfaces 


Granular details about vCPU models, their capabilities & more: 
query-cpu-definitions 
query-cpu-model-expansion 
query-hotpluggable-cpus 
query-cpus-fast, device_{add,del} 


və libvirtd caches some of this data under 
/var/cache/libvirt/qemu/capabilities/ 
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(Ə) redhat 
Run-time: Probe QEMU for CPU model specifics 


[Upstream-QEMU]$ ./qmp-shell -v -p /tmp/qmp-sock 
(QEMU) query-cpu-definitions 


"return": [ 

{ "typename": "Westmere-IBRS-x86_64-cpu", 
"unavailable-features": [], 
"migration-safe": true, 

"Static": false, 
"name": "Westmere-IBRS" }] 
. # Snip other CPU variants 


15 / 38 


we redhat 


Part Il 
CPU modes, models and flags 
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су redhat. 
Host passthrough 


Exposes the host CPU model, features, etc. as-is to the VM 


$ qemu-system-x86 64 -cpu host [...] 
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Host passthrough 


Exposes the host CPU model, features, etc. as-is to the VM 
$ qemu-system-x86_64 -cpu host [...] 


Caveats: 


No guarantee of a stable CPU for the guest 


Live migration is a no go with mixed host CPUs 





ә Most performant; ideal if live migration is not required 


O redhat. 


Host passthrough — when else to use it? 


Data Center (Intel host CPUs) 


Broadwell Broadwell Broadwell Broadwell 


Broadwell Broadwell Broadwell Broadwell 
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Host passthrough — when else to use it? 


Data Center (Intel host CPUs) 


Broadwell Broadwell Broadwell Broadwell 


Broadwell Broadwell Broadwell Broadwell 





və Along with identical CPUs, identical kernel and 
microcode are a must for VM live migration! 


(Ə) redhat 
QEMU’s named CPU models (a) 


Virtual CPUs typically model physical CPUs 
Add or remove CPU features: 


$ qemu-system-x86 64 -cpu Broadwell-IBRS,\ 
vme=on,f16c=on,rdrand=on, \ 
tsc_adjust=on, xsaveopt=on, \ 
hypervisor=on,arat=off, \ 
pdpeigb=on,abm=on [...] 
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(Ə) redhat 
QEMU’s named CPU models (a) 


Virtual CPUs typically model physical CPUs 
Add or remove CPU features: 


$ qemu-system-x86 64 -cpu Broadwell-IBRS,\ 
vme=on,f16c=on,rdrand=on, \ 
tsc_adjust=on, xsaveopt=on, \ 
hypervisor=on,arat=off, \ 
pdpeigb=on,abm=on [...] 


~> More flexible in live migration than ‘host passthrough’ 
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(Ə) redhat 
QEMU’s named CPU models (b) 


QEMU is built with a number of pre-defined models: 


$ qemu-system-x86_64 -cpu help 
Available CPUs: 


x86 Broadvell-IBRS Intel Core Processor (Broadwell, IBRS) 
x86 EPYC AMD EPYC Processor 

x86 EPYC-IBPB AMD EPYC Processor (with IBPB) 

x86 Haswell Intel Core Processor (Haswell) 


Recognized CPUID flags: 
amd-ssbd apic arat arch-capabilities avx avx2 avx512-4fmaps 
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‘host-model’ — a libvirt abstraction 


Tackles a fevv problems: 


Maximum possible CPU features from the host 
Live migration compatibility-—vvith caveats 
Auto-adds critical guest CPU flags (e.g. spec-ctr1) 
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‘host-model’ — a libvirt abstraction 


Tackles a fevv problems: 


Maximum possible CPU features from the host 
Live migration compatibility-—vvith caveats 


Auto-adds critical guest CPU flags (e.g. spec-ctr1); 
provided—microcode, kernel, QEMU & libvirt are updated! 


və Targets for the best of ‘host passthrough’ and 
named CPU models 


O redhat. 


‘host-model’ — example libvirt config 


From a libvirt guest definition: 
<cpu mode=’host-model’> 
<feature policy=’require’ name=’vmx’/> 
<feature policy=’disable’ name=’pdpelgb’/> 
</cpu> 
~> libvirt will translate it into a suitable CPU model; 
based on: /usr/share/libvirt/cpu_map/*.xml 


(Ə) redhat 
‘host-model’ and live migration 
As done by libvirt: 


Source vCPU definition is transferred as-is to the target 


On target: Migrated guest sees the same vCPU model 
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“host-model” and live migration 


As done by libvirt: 
Source vCPU definition is transferred as-is to the target 
On target: Migrated guest sees the same vCPU model 
But: When the guest ‘cold boots’, it may pick up extra 


CPU features—prevents migrating back to the source 


və Use host-model, if live migration in both directions 
is not a requirement 





(Ə) redhat 
OpenStack Nova and CPU models 


Provides relevant config attributes: 
cpu mode 
Can be: custom, host-passthrough, or host-model 
cpu model & cpu model extra flags 


Refer to libvirt’s /usr/share/libvirt/cpu пар/ж.хп1 
Or QEMU s: qemu-system-x86 64 -cpu help 


~> Details in documentation of the above config attributes 
https: //docs.openstack.org/nova/rocky/configuration/config.html 
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Part III 
Choosing CPU models & features 


(Ə) redhat 
Finding compatible CPU models 


Data Center (Intel host CPUs) 


Haswell Westmere İvyBridge SandyBridge 


Nehalem Broadwell Westmere Nehalem-IBRS 





(Ə) redhat 
Finding compatible CPU models 


Problem: Determine a compatible model among CPU variants 


(Ə) redhat 
Finding compatible CPU models 


Problem: Determine a compatible model among CPU variants 


Enter libvirt”s APIs: 
compareCPU() and baselineCPU() 
compareHypervisorCPU() and baselineHypervisorCPU() 


aS 
(New in libvirt 4.4.0) 
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O redhat. 


Intersection between these two host CPUs? 


$ cat Multiple-Host-CPUs.xml 


<cpu mode“” custom” match=’exact’> 


<model fallback=’ forbid’ >Haswell-noTSX-IBRS</model> 
<vendor>Intel</vendor> 


<feature policy=’require’ name=’vmx’/> 
<feature policy=’require’ name=’rdrand’/> 
</cpu> 
<== Sexcoucl CRY === 
<cpu mode=’custom’ match=’exact’> 


<model fallback=’ forbid’ >Skylake-Client-IBRS</model> 
<vendor>Intel</vendor> 


<feature policy=’disable’ name=’pdpetgb’/> 
<feature policy=’disable’ name=’pcid’/> 
</cpu> 
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Intersection between these two host CPUs? 


$ cat Multiple-Host-CPUs.xml 


<cpu mode“” custom” match=’exact’> 


<model fallback=’ forbid’ >Haswell-noTSX-IBRS</model> 
<vendor>Intel</vendor> 


<feature policy=’require’ name=’vmx’/> 


<feature policy=’require’ name=’rdrand’/> Two CPU 
</cpu> models 
KE БосопаиеРуЈ ==> 


<cpu mode=’custom’ match=’exact’> 


<model fallback=’ forbid’ >Skylake-Client-IBRS</model> 
<vendor>Intel</vendor> 


<feature policy=’disable’ name=’pdpetgb’/> 
<feature policy=’disable’ name=’pcid’/> 
</cpu> 
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Use baselineHypervisorCPU() to determine it 


$ virsh hypervisor-cpu-baseline Multiple-Host-CPUs.xml 
<cpu mode“” custom” match=’exact’> 
<model fallback=’ forbid’ >Haswell-noTSX-IBRS</model> 
<vendor>Intel</vendor> 
<feature policy=’require’ name=’rdrand’/> 
<feature policy=’disable’ name=’pcid’/> 
</cpu> 
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су redhat. 


Use baselineHypervisorCPU() to determine it 


$ virsh hypervisor-cpu-baseline Multiple-Host-CPUs. xml 


<cpu mode=’custom’ match=’exact’> 
<model fallback=’ forbid’ >Haswell-noTSX-IBRS</model> 
<vendor>Intel</vendor> 
<feature policy=’require’ name=’rdrand’/> 
<feature policy=’disable’ name=’pcid’/> 


</cpu> : 
P Intersection between our 


Haswell & Skylake variants 
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O redhat. 


Use baselineHypervisorCPU() to determine it 


$ virsh hypervisor-cpu-baseline Multiple-Host-CPUs.xml 
<cpu mode“” custom” match=’exact’> 
<model fallback=’ forbid’ >Haswell-noTSX-IBRS</model> 
<vendor>Intel</vendor> 
<feature policy=’require’ name=’rdrand’/> 
<feature policy=’disable’ name=’pcid’/> 
</cpu> 


və A “baseline” model that permits live migration 
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(Ə) redhat 
x86: QEMU”s “machine types” 
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(Ə) redhat 
x86: QEMU”s “machine types” 


Two main purposes: 


Emulate different chipsets (and related devices)—e.g. İntel”s 
1440FX (a.k.a “рс?) and Q35 


Provide stable guest ABI—virtual hardware remains the same, 
regardless of changes in host software or hardware 
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x86: QEMU”s “machine types” — versioned 


$ qemu-system-x86_64 -machine help 


pe 
рс-14401х-3.0 
pc-i440fx-2.9 


Standard PC 
Standard PC 
Standard PC 


Standard PC 
Standard PC 
Standard PC 
Standard PC 


(4440FX + PIIX, 1996) (alias of pc-i440fx-3.0) 
(4440FX + PIIX, 1996) (default) 
(4440FX + PIIX, 1996) 


(Q35 + ICH9, 2009) (alias of pc-q35-3.0) 
(Q35 + ICH9, 2009) 
(Q35 + ICH9, 2009) 
(Q35 + ICH9, 2009) 
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x86: QEMU”s “machine types” — versioned 


$ qemu-system-x86_64 -machine help 


pe Standard PC 
pc-i440fx-3.0 Standard PC 


– Standard РС 
Traditional 


q35 Standard PC 
pc-q35-3.0 Standard PC 
pc-q35-2.9 Standard PC 

2.8 Standard PC 


(4440FX + PIIX, 1996) (alias of pc-i440fx-3.0) 
(4440FX + PIIX, 1996) (default) 
(4440FX + PIIX, 1996) 


(Q35 + ICH9, 2009) (alias of pc-q35-3.0) 
(Q35 + ICH9, 2009) 
(Q35 + ICH9, 2009) 
(Q35 + ICH9, 2009) 
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(Ə) redhat 
x86: QEMU”s “machine types” — versioned 


$ qemu-system-x86_64 -machine help 


pe Standard PC (i440FX + PIIX, 1996) (alias of pc-i440fx-3.0) 

pc-i440fx-3.0 Standard PC (i440FX + PIIX, 1996) (default) 

pc-i440fx-2.9 Standard PC (i440FX 4 PIIX, 1996) 

q35 Standard PC (Q35 + ICH9, 2009) (alias of pc-q35-3.0) 
standard PC (Q35 + ICH9, 2009) 

| Recommended [iene PC (Q35 + ICH9, 2009) 


pc-q35-2.8 Standard PC (Q35 + ICH9, 2009) 


~> Versioned machine types provide stable guest ABI 
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(Ə) redhat 
Machine types and CPU features 


Changing machine types is guest-visible 
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O redhat. 


Machine types and CPU features 


Changing machine types is guest-visible 
After a QEMU upgrade, when using libvirt: 


Need an explicit request for machine type upgrade 


The guest needs a ‘cold-reboot’ (i.e. an explicit stop + 
start)—to allow QEMU to re-exec() 


və Change machine types only after guest workload 
evaluation—CPU features & devices can differ 


(Ə) redhat 
x86: Recommended guest CPU models 


Before configuring guest CPUs: 


Update microcode, host & guest kernels; refer 
to—/sys/devices/system/cpu/vulnerabilities/ 
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x86: Recommended guest CPU models 


Before configuring guest CPUs: 


Update microcode, host & guest kernels; refer 
to—/sys/devices/system/cpu/vulnerabilities/ 


Update libvirt & QEMU—and explicitly update guest 
CPUs to patched variants (e.g. the *-IBRS models) 


Cold-reboot the guests—to pick up new CPUID bits 


33 / 38 


(Ə) redhat 
x86: Recommended guest CPU models 


Before configuring guest CPUs: 


Update microcode, host & guest kernels; refer 
to—/sys/devices/system/cpu/vulnerabilities/ 


Update libvirt & QEMU—and explicitly update guest 
CPUs to patched variants (e.g. the *-IBRS models) 


Cold-reboot the guests—to pick up new CPUID bits 
və Guidance: qemu/docs/qemu-cpu-models.texi 


(Thanks, Daniel Berrange) 
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(Ə) redhat 
x86: Important CPU flags 


To mitigate guests from multiple Spectre & Meltdown variants: 


Intel: ssbd, pcid, spec-ctrl 
AMD: virt-ssbd, amd-ssbd, amd-no-ssb, ibpb 


Some are built into QEMU”s x-IBRS & *-IBPB CPU models 
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(Ə) redhat 
x86: Important CPU flags 


To mitigate guests from multiple Spectre & Meltdown variants: 


Intel: ssbd, pcid, spec-ctrl 
AMD: virt-ssbd, amd-ssbd, amd-no-ssb, ibpb 


Some are built into QEMU”s x-IBRS & x-IBPB CPU models 


~~ Details: 
qemu/docs/qemu-cpu-models.texi 
https://uvv.qemu.org/2018/02/14/qemu-2-11-1-and-spectre-update 
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O redhat. 


Future ‘expectations’ from applications? 


“QEMU and libvirt took the joint decision 

to stop adding new named CPU models when 
CPU vulnerabilities are discovered from this point 
forwards. Applications / users would be 

expected to turn on CPU features explicitly as 
needed and are considered broken if they don t 
provide this functionality.” 


— “CPU model versioning separate from machine type versioning” 
From “qemu-devel” mailing list 


35 / 38 


су redhat. 


References 


E 





CPU model configuration for QEMU/KVM x86 hosts, by Daniel Berrangé 


https: //www.berrange.com/posts/2018/06/29/cpu-model-configuration-for-qemu-kvm-on-x86-hosts 


Mitigating Spectre and Meltdown (and (1ТЕ), by David Woodhouse 


https: //kernel-recipes.org/en/2018/talks/mitigating-spectre-and-meltdown-vulnerabilities/ 


Exploiting modern microarchitectures—Meltdown, Spectre, and other 
hardware attacks, by Jon Masters 


https: //archive.fosdem. org/2018/schedule/event/closing_keynote 


KVM and CPU feature enablement, by Eduardo Habkost 


https: //wiki.qemu.org/images/c/c8/Cpu-models-and-libvirt-devconf-2014. pdf 


36 / 38 


“ав redhat. 


Questions? 


E-mail: kashyap@redhat . соп 
IRC: kashyap — Freenode & OFTC 
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(Ə) redhat 
Related talks at the KVM Forum 


(1) Security in QEMU: How Virtual Machines Provide 
Isolation — by Stefan Hajnoczi 


— Happening now, but it’s being recorded 


(2) What Did Spectre and Meltdown Teach about CPU 
Models? — by Paolo Bonzini 
— 26-OCT, Wednesday: 11:30 — 12:00 
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